Virtual Reality (VR) is a mobile or computer simulated environment that gives the user a sense of immersion and presence through three-dimensional (3D) images and the support of visual, auditory and/or tactile feedback. Augmented Reality (AR) is an overlay of computer-generated contents (e.g. sound, video, graphics, GPS data) on the real world where, although in it, these contents do not and/or cannot interact with this world. Mixed Reality (MR) is almost the same as AR, but the created content is anchored and interacts directly with the world in real time and vice-versa. These technologies are all part of an emerging world of Virtual Environments (VEs), but the most prominent is virtual reality.
Although part of science fiction and a technology desired for decades, virtual reality as we know today arised after a 2010 prototype and 2012 Kickstarter campaign that would later become the PC-powered Oculus Rift. Hospitals and research labs have already been used VR for more than 20 years by then, but the costs and structure required to maintain it are difficult to reach for consumer-level users.
In the past five years, the technology behind VR and its multiple applications has advanced quickly. Several competitors have emerged since the Oculus Rift, from HTC Vive® to gaming console-powered PlayStation® VR and mobile headsets like Google Daydream® and Samsung Gear® VR. Through these companies, not only affordable priced hardware has entered the market, but also new technologies and interaction approaches were produced.
Developers are directing their efforts to testing and creating applications, games, experiences and exploring the next features that this technology can offer. At the same time, the technology has gathered the interest of a number of professionals in an attempt to handle existing problems on the hardware and also pursuing better and detailed studies regarding the current shortcomings/challenges of the virtual reality applications. The design of appropriate 3D interaction techniques in virtual environments is one of these challenges.
The act of selecting an object in a 2D interface is a familiar interaction that normally involves an input device (often a mouse) that when moved will affect the position of a virtually created cursor. Numerous hardware and operating system enhancements were applied over the years to make it as accurate and seamless as possible. In 3D environments, although having the positive side of allowing multiple selection styles and freedom of movement, this is a more demanding task that requires a great deal of dexterity. This issue is further increased by the fact that, even in the most sophisticated virtual environments, it is still difficult for a user to accurately understand and define object positioning and spatial limits. Accurate object selection in VEs can also be affected by how the interfaces are created/organized and tasks such as using a scroll bar, manipulating timelines or trying to make other kinds of fine selections can be too demanding with the use of available controls.
Since object selection is one of the most common and fundamental interactions in VEs, a poorly designed interaction tool can lead to an overall bad user performance/experience even in the most refined applications. Past studies have revealed that virtual pointing techniques result in better selection effectiveness compared to other 3D methods, so the disclosed matter aims to create methods and components that can improve current virtual pointers and make user tasks easier to perform.
In the prior art, the following technologies that enable the implementation (technical viability) of the present invention can be found:
1. Nearly all digital interfaces today are graphical user interfaces (GUIs). An interface is a set of data, commands/controls and/or menus/navigation displayed on a screen, through which a user can communicate with a program—in other words, GUI is the part of a system through which the user will interact. The first commercially available GUI was released in 1981 and since then it kept evolving to adjust to new technologies and types of digital devices currently available.
2. GUIs are affected by and can be controlled by a number of input devices/methods, such as (but not limited to): mouse/keyboard, controllers, touchscreens and motion detection.
3. Most of the existing input devices/methods are based on the idea of positioning its virtual representation on a display. For the sake of clarity, this representation is referred to as pointer.
4. Head-mounted display (HMD) is a device that is used over the head and that features a display in front of one or both user's eyes. This display streams data, images and other types of information. Furthermore, HMDs can have a display for each eye (e.g. Oculus Rift) or one display for both eyes (e.g. Samsung Gear VR). Basically, HMD is a content-viewing hardware. Initially created for the aviation industry, the HMD is used as the main hardware in the VR/AR industry. Types of HMD include: Slide-on HMD (mobile-based, uses smartphone as display, for processing and tracking), Tethered HMD (PC-based, provides everything aside from processing, which is done by the PC) and Integrated HMD (totally independent device, containing everything for delivering an immersive experience).
The patent document CN105498213A titled “Aiming method and device in shooting game”, published on Apr. 20, 2016, by HANGZHOU WIZARD GAME TECH CO LTD, proposes the use of targeting techniques for shooting games, comprising of a terminal with touch screen displays, which behavior varies according to sight position and creates an amplification of targeting region that ultimately creates an enlarged area to perform shooting operations. Although this invention seems to serve as an assistance for shooting operations, it differs from the present proposal because of its use cases, equipment, and the method itself: VR pointers are not limited to shooting game operations, serving to any VR application interface that offers aiming, clicking, and dragging tasks; VR technology does not use display capable touchscreens, but full 360° rotation capable touchpads and buttons that integrate with head mounted displays; and finally, our proposed invention uses distinct methods that perform pointer stability, assisted aiming, and resistive drags.
The patent document WO2017051985, titled “Haptic simulating device”, published on Mar. 30, 2017, by KANG TOO HWAN, discloses a solution to be applied on simulator devices (especially hardware and peripherals related to Virtual Reality) for enhancing the users' immersion and presence through a specific haptic feedback. The invention is an evolution of that they call conventional techniques for creating the feeling of real movement/force rotational force, slip and vibration are all generated through it for this purpose. Furthermore, the invention can be implemented in small devices (such as a game controller) and larger ones (such as a bicycle handle), because its goal is to be used for a number of purposes. Although it is related to VR and controllers, document WO2017051985 differs from the present invention because it defines behaviors and construction models for hardware products in its claims instead of how the virtual pointer movement would be processed.
The patent document US20100009733A1, titled “Game aim assist”, published on Jan. 14, 2010, by Sony Comp Entertainment US, refers to assisted aiming in electronic games. The idea is based mostly on games in third person view wherein the interface between the game player and the game character is made through a controller with analog stick (joystick). The created method makes it easier for a player to aim at objects/enemies—especially when in great number—through the use of a user-defined focus area, and other points such as a method for choosing targets according to the distance between game character and objects. Although it directly relates to an aim assist area, it differs from the present invention because this method is directed at gaming interfaces and the idea of aiming while using an in-game weapon and a real-world analog stick, while also focusing on the third-person view (where a game character is placed in the middle of the screen and other interaction points appear around it). These differences become clear as pointed below:
a. On claims 3 and 23, document US20100009733A1 describes about a method based on an input from a single analog control stick but the present invention proposes methods that do not limit the controllers model/format, only indicating that it must be a VR/AR related controller;
b. On claims 12 and 13, document US20100009733A1 describes methods for adjusting the aim and focus area depending on a weapon being operated by a game character but the present invention does not limit the methods to scenarios where characters and weapons are available;
c. On claim 15, document US20100009733A1 describes a user-customizable focus area but the present invention is not based on user input—instead, it is based on the interface.
The patent document US20060033712A1, titled “Displaying visually correct pointer movements on a multi-monitor display system”, published on Feb. 16, 2006, by “Microsoft Corp”, is focused in creating a solution for moving a pointer across monitor boundaries in a correct manner, including situations where monitors are in different settings (position), and have different sizes and resolutions. The document US20060033712A1 comprises a multi-monitor computer system, a display surface and a virtual space. The method establishes a virtual space for the multi-monitor computer system with a plurality of virtual display areas wherein these displays are arranged in the virtual space reflecting the visual appearance of how the monitors appear to the user (according to a previously done calibration). This calibration also gathers information about relative screen resolution and pixel resolution. In the end, the pointer is moved, and its representation adapted to a seamless movement to be displayed in the user's monitors. Although it is directly related to pointers and it allows any type of hardware input, document US20060033712A1 differs from the present invention because it deals directly with multi-screen systems and it focus in moving the pointer in a correct manner between these screens.
The patent document U.S. Pat. No. 5,808,601A, titled “Interactive object selection pointer method and apparatus”, published on Sep. 15, 1998, by IBM, focuses on providing an improved selection pointer control method in which the selection pointer is drawn towards selectable items when near enough, according to a mathematical process. It further describes a system in which its mouse (or other type of control hardware) has two identities: real and virtual. The idea was to make it easier for users to select items on small interfaces and displays by having a method of facilitated pointing/selection. The behavior and execution in document U.S. Pat. No. 5,808,601A are very similar to the aim assist method of the present invention; however, its difference lies in the solution's mathematical process and results. Furthermore, the present invention comprises the combination of 3 different methods in order to achieve a better pointer interaction instead of relying only on the “magnet” aspect.
a. On claim 1, the patent document U.S. Pat. No. 5,808,601A describes the use of the center of the object to determine the distance from the pointer to it by checking if the magnitude of the value calculated is greater or equal to the D value. However, the present invention proposes a different approach wherein a specific position on the main area is calculated to find a relative position on the specified area—this means that we calculate our relative position based on the middle of each direction (vertical/horizontal) but not from the center itself. In the same way, in the method of the present invention it is possible to set the aim assist behavior based on each side/direction of the object (left, right, top or bottom) and this behavior will happen independently for each individual direction.
b. On claim 4, the patent document U.S. Pat. No. 5,808,601A describes a pointer that is independent of the actual physical location of the controller, but in the present invention the pointer depends completely on its physical position over an object to determine a relative position in the specified area.
Non-patent document Poster, titled “The Flexible Pointer: An Interaction Technique for Selection in Augmented and Virtual Reality” by A. Olwal and S. Feiner (available online at https://uist.acm.org/posterexamplegallery/pointer.pdf, last access on July 2017) describes a virtual flexible pointer that allows a user to point and indicate objects more clearly, even if partially hidden/obscured. While using hand position and orientation to control the pointer's length and curvature, this invention uses human behavior as reference for its construction—the inventors' informal observations revealed that users do not always point straight when indicating objects of interest, especially when pointing to something that is not currently visible, and these curved gestures are often combined with movement. Despite using the same curved representation, this solution differs from the present invention because the curvature used in our ray is used for representing resistive drags—the curve only occurs when the user holds and drags an interface element instead of being used for normal selection.